AI Guardrails Index:

We broke AI guardrails down to six categories.

We curated datasets and models that demonstrate the state of AI safety using LLMs and other open source models.

Introduction

PII redacting guardrails are crucial for AI applications across sensitive industries. They protect personal data in financial services, healthcare, HR, legal, and call centers by automatically masking sensitive information. This ensures compliance with regulations like GDPR, CCPA, and HIPAA, maintains client confidentiality, and protects employee privacy. Implementing these guardrails allows organizations to leverage AI while safeguarding user trust and sensitive information.

Results

Leaderboard

Metric:

Task:

Developer	Model	Latency	Outcome
Urchade Zaratiana	GLiNER for PII	0.05 ms	72.04%
Guardrails AI	Guardrails PII	0.07 ms	74.80%
Microsoft	Presidio	0.01 ms	39.05%

Dataset Breakdown

Developer	Samples
TIME	24022
USERNAME	17209
IDCARD	16693
EMAIL	15883
SOCIALNUMBER	15791
PASSPORT	15570
DRIVERLICENSE	15207
BOD	14531
LASTNAME1	14228
IP	14111
SEX	12989
GIVENNAME1	12602
TEL	12601
CITY	12364
POSTCODE	12160
STATE	11975
STREET	11963
BUILDING	11835
TITLE	11097
DATE	10734
COUNTRY	10694
PASS	10303
SECADDRESS	5066
LASTNAME2	3712
GIVENNAME2	3358
GEOCOORD	1285
LASTNAME3	1190
CARDISSUER	17

See the full dataset here: PII Detection dataset

Conclusion

Azure's Presidio Detect PII showed a surprisingly low recall (0.3905) despite a good precision. Gliner PII, on the other hand, offers better recall at the cost of lower precision. Guardrails AI combines the strengths of both, achieving the highest F1 score with improved recall while maintaining high precision. Diving deep into different PII types, we discovered that Guardrails AI shows comprehensive capabilities across various PII categories, including improved detection of unstructured PII for which Azure's Presidio Detect PII was struggling. Guardrails AI also offers fast processing speeds (0.6526s on CPU and 0.0695s on GPU ) making it suitable for both bulk processing and real-time applications. This balanced approach provides a robust solution for data protection and compliance needs across different scenarios.